Image processing apparatus and method

ABSTRACT

An apparatus having image processing function performs image processing method. In the image processing method, a first portion of data is acquired as a data stream obtained from image data that has been sequentially converted and encoded, then the acquired data is decoded to obtain a two-dimensional image. The two-dimensional image is analyzed to determine a region of interest within the two-dimensional image. Then, a second portion of data is acquired as a data stream obtained from the image data based on the region of interest. Thus, areas that are useful to the diagnosis can be displayed promptly and in detail.

FIELD OF THE INVENTION

The present invention relates to an apparatus and method having an imageprocessing function, and more particularly, to an apparatus and methodhaving an image processing function for medical X-ray images.

BACKGROUND OF THE INVENTION

It is well known that certain types of fluorescent materials, whenexposed to radiation—whether X rays, alpha rays, beta rays, gamma rays,electron beams or ultraviolet light—store a certain portion of theenergy with which they are irradiated so that, when irradiated withvisible or other excitation light, these fluorescent materials undergoaccelerated phosphorescence. Such materials are called photostimulablephosphors.

In Japanese Laid-Open Patent Application Nos. 55-12429, 56-11395 and thelike, radiation image information recording and reproduction systemsthat utilize this photostimulable phosphors have been proposed.According to such systems, radiation image information from a human bodyor other object exposed to radiation is first recorded on aphotostimulable phosphor sheet. The photostimulable phosphor sheet isthen exposed to a laser beam or other excitation light, causing thephotostimulable phosphor sheet to undergo accelerated phosphorescence.The accelerated phosphorescence is then read photoelectrically toacquire image signals, which are then used to make visible a radiationimage of the irradiated object by displaying it on a photosensitiverecording medium, CRT or the like.

Recently, apparatuses that use semiconductor sensors similarly to takeX-ray images have been developed. Systems such as these have thepractical advantage of being able to record images over a considerablywider range of radiation exposures than radiation photo systems usingsilver chloride film can do. In other words, such systems can providevisible radiation images unaffected by fluctuations in radiationexposure amounts, by acquiring X rays over an extremely broad dynamicrange using photoelectric conversion means, converting the acquired Xrays into electrical signals, and using the electrical signals to outputa visible image to a photosensitive recording medium, CRT or otherdisplay apparatus.

The electrical signals acquired as described above can be converted intodigital information, and the digital information so obtained can bestored in a storage medium such as a memory. This type of digitalinformation can then be supplied to an information processing apparatusfor digital image processing, in order to provide various types ofdiagnostic support.

However, X-ray images contain a large amount of information, so storingand transmitting such images involves very large volumes of data.Advanced encoding is used to reduce the tremendous volumes of datainvolves in storing and receiving such X-ray images, by eliminating theimage redundancy or by altering the image content so as to degrade theimage by an amount still not easily discernible to the naked eye.

Thus, for example, in the Joint Photographic Experts Group (JPEG)standard recommended by the International Standard Organization (ISO)and International Telecommunication Union (ITU) as the internationalstandard coding method for still pictures, Differential Pulse CodeModulation (DPCM) is used for reversible compression and Discrete CosineTransform (DCT) is used for non-reversible compression. A detailedexplanation of JPEG is provided in ITU-T Recommendation T.81, ISO/IEC(International Electrotechnical Commission) 10918-1 and so will beomitted here.

Additionally, much recent research has concentrated on compressionmethods using Discrete Wavelet Transform (DWT). The advantage of DWT isthat the blocking artifacts seen with DCT do not appear.

On the other hand, it is possible to improve the efficiency ofcompression when compressing an X-ray image by determining an area ofinterest within the image and reducing the data compression ratio forthat area so as to provide superior picture quality for it. Also, whenperforming lossless coding as well, it is possible to priority code anarea of interest (hereinafter sometimes referred to as AOI) and thenpriority decode that same area when reading out the image. However, thedetermination of an AOI is not an easy choice to make, involving as itdoes a medical judgment.

With these considerations in mind, the applicant has proposed a method(and apparatus) that, when an image is compressed, analyzes an inputimage to extract the X-ray radiation field area, then further extractsan X-ray pass-through region from the extracted radiation field area,sets the area of interest as that part of the radiation field area otherthan the pass-through region, level-shifts and codes the imagecorresponding to the area of interest so as to priority code the area ofinterest. The problem with this method is that the pass-through regionwithin the carved-out image (that is, the radiation field area)constitutes approximately 20 percent of the field, so in order toimprove the data compression ratio the AOI must be narrowed further.

Additionally, there are many instances in which it is useful to set thearea of interest when displaying the contents of the compressed imagefile. For example, displaying one or more images composed of 2,000×2,000pixels on a monitor of 1,000×1,000 pixels involves a choice betweeneither displaying only a portion of one image or displaying reducedimage(s). Of course, it is possible to display the entire unreducedimage and simply scroll down the screen using a mouse or a track ball,but such an expedient will not achieve the objective of simultaneouslydisplaying a plurality of images. It is possible to display only the AOIif the AOI is set for the images when the images are being filed, but ifthere is no AOI, or if the AOI has been set but overflows its allottedspace, in either case it is necessary to use a reduced display.

A reduced display creates its own problems, insofar as such display haspoor spatial reduction capabilities, which can make detailed inspectionof an image impossible. Techniques have been proposed to compensate forthis defect, for example, use of a normalized rectangle or circularmagnifying lens that an operator (a physician) moves across the top ofthe CRT so as to partially display the original subject. However, theproblem remains as to whether or not it is possible to selectappropriately an area to enlarge, given that such display has poorspatial resolution capabilities.

It is also sometimes the case that X-ray images are encoded beforestorage so as to reduce the volume of data involved, but it ispreferable that such encoded images be displayed promptly, withoutdelay. The ability to display such encoded images promptly makes itpossible also to priority display an area useful to the diagnosis fromthe encoded image data.

SUMMARY OF THE INVENTION

Accordingly, the present invention has as its object to provide anapparatus and a method having an image processing function that solvethe above-described problems of the conventional art.

Another and more specific object of the present invention is to make itpossible to expedite display of an area useful to a diagnosis fromencoded image data.

Still another and more specific object of the present invention is tomake it possible to expedite reading of an area of interest (AOI)important to the diagnosis or observation as determined by a diagnosticsupport means and to improve the image quality of that area, so as toefficiently improve the accuracy of diagnostic support.

Still another, further and more specific object of the present inventionis to make it possible to acquire image data with an image qualitysuited to the objective of the diagnostic support and to achieveefficient image transmission.

The above-described objects of the present invention are achieved by animage processing apparatus comprising:

first acquisition means for acquiring a first portion of a data streamobtained from image data that has been sequenced, converted and encoded;

decoding means for decoding the data stream acquired by the firstacquisition means and obtaining a two-dimensional image;

analysis means for analyzing the two-dimensional image obtained by thedecoding means and determining an area of interest within thetwo-dimensional image; and

second acquisition means for acquiring a second portion selected fromthe data stream based on the area of interest determined by the analysismeans.

Additionally, the above-described objects of the present invention arealso achieved by an image processing method, comprising:

a first acquisition step for acquiring a first portion of a data streamobtained from image data that has been sequenced, converted and encoded;

a decoding step for decoding the data stream acquired in the firstacquisition step and obtaining a two-dimensional image;

an analysis step for analyzing the two-dimensional image obtained in thedecoding step and determining an area of interest within thetwo-dimensional image; and

a second acquisition step for acquiring a second portion selected fromthe data stream based on the area of interest determined in the analysisstep.

According to the above-described inventions, priority display an areauseful to a diagnosis from encoded image data can be achieved.

Additionally, the above-described inventions make it possible topriority read an area of interest (AOI) important to the diagnosis orobservation as determined by a diagnostic support means and to improvethe quality of that image of that area, so as to effectively improve theaccuracy of diagnostic support.

Additionally, the above-described inventions make it possible to acquireimage data of an image quality suited to the objective of the diagnosticsupport and to achieve efficient image transmission.

Other objects, features and advantages of the present invention willbecome more apparent from the following description taken in conjunctionwith the accompanying drawings, in which like reference charactersdesignate the same or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram of an image processing apparatus according toone embodiment of the present invention;

FIG. 2 is a flow chart illustrating operations of an image processingapparatus according to one embodiment of the present invention;

FIG. 3 is a flow chart illustrating steps in a shadow-based diagnosticsupport process;

FIG. 4 is a block diagram showing an internal composition of a shadowextraction unit;

FIGS. 5A, 5B and 5C are diagrams illustrating characteristics of theshadow extraction unit;

FIG. 6 is a diagram showing a neural network for determining whether theextracted area is or is not a positive test result;

FIGS. 7A, 7B, 7C and 7D are diagrams illustrating a hypothetical exampleof the shadow-based diagnostic support process;

FIG. 8 is a flow chart illustrating steps in a texture diseasediagnostic support process;

FIG. 9 is a block diagram of a texture disease diagnostic support unit;

FIG. 10 is a block diagram of a segmentation unit of an image processingapparatus according to the present invention;

FIG. 11 is a diagram showing the functional composition of thesegmentation unit during a training phase;

FIG. 12 is a diagram showing the functional composition of thesegmentation unit during a utilization phase;

FIGS. 13A, 13B and 13C illustrate a process of segmentation;

FIGS. 14A, 14B and 14C are specific examples of hypothetical X-rayimages;

FIG. 15 is a diagram illustrating a process of dynamically generating acode sequence between an image input unit and an image server;

FIG. 16 is a diagram illustrating a process of code sequencetransmission where an image is divided into a tile and encoded;

FIG. 17 is a diagram illustrating a process of code sequencetransmission where an image has been coded using DWT;

FIG. 18 is a diagram illustrating a hypothetical layering of a JPEG2000code sequence;

FIG. 19 is a diagram illustrating encoded data arranged in layers;

FIG. 20 is a block diagram of an image processing apparatus according toan embodiment of the present invention; and

FIGS. 21A and 21B are diagrams illustrating the relation between codeblock and area of interest.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail, with reference to the accompanying drawings.

According to the following embodiments, an image processing apparatuswhich, upon reading coded X-ray medical image from a storage medium,automatically determines effective regions for diagnosis, reads datacorresponding to the determined regions by priority, and displays theread data, will be explained.

FIG. 1 is a block diagram for illustrating in general outline an imageprocessing method as executed by an image processing apparatus accordingto one embodiment of the present invention. FIG. 20 is a is a blockdiagram of an image processing apparatus according to an embodiment ofthe present invention, designed to execute the image processing methoddescribed in FIG. 1. For convenience of explanation, a description willfirst be given of the image processing apparatus.

As shown in FIG. 20, the image processing apparatus according to oneembodiment of the present invention comprises a central processing unit(CPU 21) that executes either a control program stored in a ROM 22 or acontrol program loaded into a RAM 23 from an external storage device.The ROM 22 stores not only the control program executed by the CPU 21but also a variety of types of data to be described later. The RAM 23provides not only a load space for the control program but also workspace for the CPU 21.

Reference numeral 24 denotes a display unit, which displays a variety ofscreens according to commands issued by the CPU 21. Reference numeral 25is an external storage device. The external storage device 25 may be ahard disk or similar storage medium. Reference numeral 26 denotes aninput device, designed to accommodate operations performed by anoperator the apparatus. The input device may be a keyboard or a mouse.Reference numeral 27 denotes a network interface, connected so as toenable communications with a network 28. The network may be a local areanetwork (LAN), the internet, or the like. Reference numeral 20 is a bus.The bus 20 connects the above-described constituent parts to each other.

Reference numeral 29 is an image server for storing encoded image data.The image server 29 provides a data stream of sequentially converted andencoded image data.

In FIG. 1, reference numeral 11 denotes an image input unit, whichpresupposes input of encoded image data. It should be noted that thedescription that follows assumes a DWT-converted image is converted intoan encoded stream and input. However, as can be appreciated by those ofordinary skill in the art, the present invention is not limited toDWT-converted images but is equally applicable to DCT and to KLsequentially converted images. In this specification, the encryption ofsource information, whether by DCT, KL conversion or DWT, iscollectively referred to as sequential transformation. It should furtherbe noted that DCT and KL conversion are common techniques, described indetail, for example, in Anil K. Jain, “Fundamentals of Digital ImageProcessing” (Prentice-Hall Inc., 1989).

The encoded image data is transmitted from the image server 29 connectedto a network. The data input by the image input unit 11 is transmittedby a decoder unit 12. An ISO/IEC 15444 (hereinafter referred to asJPEG2000) is here used for the decoder unit 12 that decodes the DWTimage, the details of which are described in detail in the publishedspecifications thereof. An image 13 that is the output from the decoderunit 12 is displayed by an image display unit 15 and also provided to adiagnostic support unit 16. Alternatively, as can be appreciated bythose of ordinary skill in the art, the image 13 may be fed directly tothe diagnostic support unit 16.

It is important to understand here that the initial screen used for thediagnostic support unit 16 may be of a picture quality that satisfiesthe minimum requirements for the purposes of the diagnosis (sometimeshereinafter referred to as the diagnostic purpose) and that it is notnecessary to read the entire data stream converted for diagnosticsupport. It should be noted that the minimum required picture qualityfor diagnostic purposes, in other words the configuration of thelayering for coding, is fixed by an initial stream setting unit 18.

It should be noted that in the following description, which uses afrontal image of the chest portion of a human being in the detection ofshaded areas (hereinafter shadows) indicative of possible tumorstherein, in this instance a relatively low-resolution reduced image issufficient for the diagnostic purpose. Accordingly, the initial streamsetting unit 18 is set so that the partial data corresponding to theimage is transmitted as the initial image. There are a plurality oftypes of diagnostic supports, and the quality of the initial image isdetermined by the type of diagnostic support to be performed by thesystem. Thus, for example, the detection of tumors, which are relativelylarge objects, does not require a very high degree of resolution, so aspatial reduction capability of approximately one fifth and a densityreduction capability of one half may be used for the picture quality ofthe initial image.

On the other hand, a shadow like that of frosted glass requires arelatively high degree of resolution, so a spatial reduction capabilityof approximately one half and a density reduction capability of onethird is set for the picture quality of the initial image. These initialsettings are closely related to parameters set in the diagnostic supportalgorithm internal, and are determined by trial and error. When an imagethat satisfies the foregoing conditions is input, the diagnostic supportunit 16 starts the diagnostic support process. If a further data streamis required during the process of diagnostic support, then data is inputfor the required area via an input control unit 14. An area 17 that isidentified by the diagnostic support unit 16 as positive (hereinafterreferred to as a positive area 17) is displayed as contour informationoverlaying the reduced image displayed on the image display unit 15 andprovided to the input control unit 14.

Summarizing the above-described operations reveals the following: FIG. 2is a flow chart illustrating operations of an image processing apparatusaccording to one embodiment of the present invention. First, as shown inFIG. 2, in a step S21 an initial stream setting unit 18 sets the initialstream appropriate to the purpose of the diagnostic support. In a stepS22, the image input unit 11 acquires encoded image data and sends thecodes image data to the decoder unit 12. At this time, the input controlunit 14 controls the image input unit 11 so that image data is input inaccordance with the settings determined by the initial stream settingunit 18.

Then in a step S23, the encoded image data input by the image input unit11 is decoded and the image data 13 obtained thereby is displayed by theimage display unit 15. In a step S24, the diagnostic support unit 16analyzes the decoded image data 13, checks the disease location on thepatient's body, and outputs positive area data 17. In a step S25, theinput control unit 14 determines which areas in the encoded image inputin step S22 should be concentrated on for further analysis (hereinaftersometimes referred to as priority displayed), based on the positive areadata 17 output from the diagnostic support unit 16. In a step S26 theprocess returns to step S22 if image data acquisition is not finished,at which point the input control unit 14 directs the image input unit 11to acquire the encoded image in accordance with the area designated bythe input control unit 14. The encoded image (or images) thus input bythe image input unit 11 is (are) then decoded in order by the decoderunit 12 and displayed by the image display unit 15. The acquisition offurther encoded image data helps to improve both the accuracy of thediagnostic support process of step S24 and the resolution of the imagedisplay in step S23.

The process described above results in the automatic selection from theencoded data of an area useful to the diagnosis, with this area ofinterest useful to the diagnosis being priority displayed for furtheranalysis. Such automated priority display of the area of interestprovides advantages similar to the prompt display of X-ray images.

A description will now be given of the diagnostic support unit 16.

Although the diagnostic support unit 16 performs different processesdepending on the purpose of the diagnostic support, basically theseprocesses can be divided into two major types: shadow-based diagnosticprocesses and texture disease diagnostic processes. The former includeprocessed used in the detection of cancerous tumors and calcification asrevealed in chest images or tumors as revealed in mammograms. The latterinclude processes used in the detection of interstitial pneumonia asrevealed in chest images.

A description will first be given of shadow-based diagnostic support.

FIG. 3 is a flow chart illustrating steps in a shadow-based diagnosticsupport process. FIG. 4 is a block diagram showing an internalcomposition of a shadow extraction unit. FIGS. 5A, 5B and 5C arediagrams illustrating characteristics of the shadow extraction unit.FIG. 6 is a diagram showing a neural network for determining whether theextracted area is a positive test result or not. FIGS. 7A, 7B, 7C and 7Dare diagrams illustrating a hypothetical example of the shadow-baseddiagnostic support process.

As shown in FIG. 3, in a step S31, decoded image data 13 is input to ashadow detection unit for detecting the presence of shadows in thedecoded image data 13.

As shown in FIG. 4, a high-frequency image and a low-frequency image areproduced from the input image data 13 using a high-pass filter 41 and alow-pass filter 42. An operation part 43 takes the difference betweenthese two images and produces shadow candidate image data 44.

Examples of the characteristics of the high-pas filter 41 and thelow-pass filter 42 are shown in FIGS. 5A and 5B. Additionally, thecharacteristics formed by the difference between the high-frequencyimage and the low-frequency image are shown in FIG. 5C, which can bethought of as a matched filter for detecting a circular pattern of agiven size from the images. In other words, as shown in FIG. 5C, afilter that extracts only an area that corresponds to a given frequencyin the image can be understood as in effect extracting only objects of agiven size from the image.

As should be clear from the foregoing description, the size of thecircular pattern to be extracted can be changed by adjusting thecharacteristics of the high-pass filter 41 and the low-pass filter 42used in the shadow extraction process described above. In general, it isvery difficult to select and extract tumors having a diameter of 5 mm orless from frontal chest images, because such an image contains a largenumber of signals of the same size and most of these signals do notindicate an illness. Accordingly, the present embodiment uses a matchedfilter of approximately 10 mm. However, those of ordinary skill in theart can appreciate that the present invention is not limited to a singlediameter of a circular pattern to be extracted by the matched filter.Rather, a plurality of different circular patterns may be used. Thus, byway of illustration only, a series of three different filters forcircular patterns of, for example, diameters of 8 mm, 12 mm, and 16 mm,may be used, with the resulting shadow candidates thus extracted beingused in later processes.

Next, in a step S32, pathological parameters are calculated for each ofthe shadow candidates extracted in the shadow extraction process S31described above. The parameters thus calculated are surface area,circularity and threshold sensitivity, each of which is described indetail as follows. It should be noted that those parameters arecalculated for each shadow candidate, and calculation of thoseparameters requires binarization of the matched filter output image.Through the experiment, the threshold value for the binarization is setto a pixel value corresponding to 10 percent accumulated frequency ofthe accumulated histogram of pixel values of the matched filter outputimage.

-   -   (1) Surface area S=[number of pixels contained in the shadow        candidate]×[surface area of one pixel]    -   (2) Circularity C=A/S, where A is the overlapping surface area        of a circle of actual diameter D and the shadow candidate when        the circle is positioned at the center of the shadow candidate.    -   (3) Threshold sensitivity=|2×S10−S5−S15|

The threshold sensitivity is the change in surface area of the shadowwhen the threshold value is changed, where S5, S10 and S15 denotechanges to the threshold value of 5, 10 and 15 percent, respectively,with the surface area “||” being expressed as an absolute value.

In a step S33, each of the shadows extracted in step S31 are determinedto be either positive or false-positive based on the parameterscalculated as described above. “False-postive” as used here simply meansthat the shadow is not “positive”, and no more. It should be noted thatalthough the present embodiment uses three parameters, the presentinvention is not limited to such a configuration but may be adapted touse more or fewer parameters as required.

Additionally, it should be noted that a determinating part comprising aneural network is used to carry out the determinations made in step S33.A description will now be offered of this determining part, suchdescription being divided between a training phase of such determiningpart and a utilization phase thereof. Those of ordinary skill in the artcan appreciate that the determining part of the present embodiment usesa neural network, the present invention is not limited to such aconfiguration but can be adapted to suit the purpose of the diagnosis,and thus, for example, and by way of illustration only, may be linearlyclassifiable. In such a case, it is acceptable to use a linearclassification that is simpler and has fewer parameters than thatdescribed above.

In the training phase, while the results of the determinations made instep S33 as described above are displayed for the shadows that have beenidentified as either positive or false-positive the parameters used tomake those determinations are input and the neural network internalcoefficients are trained. A wide variety of systems have been developedas neural networks. In the present invention, the neural network may bea feed forward-type error reversal neural network such as that developedby D. E. Rumelhart and described in D. E. Rumelhart and J. L. McCleland,“Parallel Distributed Processing: Explorations in the Microstructure ofCognition, Vol. 1: Foundation” (Cambridge: The MIT Press, 1986.Alternatively, the neural network may be a Radial Basis Function neuralnetwork (hereinafter abbreviated as RBF-NN) such as that described in C.Bishop, “Improving he Generalization Properties of Radial Basis FunctionNeural Networks”, Neural Comp., Vol. 3, pp. 579–588, 1991.

As shown in FIG. 6, the present embodiment uses a triple input nodeRBF-NN, which has a three-layer structure like that of the Feed Forwardtype. The three layers are an input layer, an intermediate layer and anoutput layer. The input layer has as many nodes as the number ofparameters extracted in the pathological parameter extraction process ofstep S32. The RBF neurons of the intermediate layer are equipped withoutput characteristics so as to have a Gauss distribution asnon-linear-type elements. The number of RBF neurons depends on thenumber of learning cases and the complexity of the problem, but in orderto set the calculation time to a reasonable period 100 is a reasonablenumber. The number of outputs of the neural network is one, with “1”output for a positive and “0” output for a negative. In the presentembodiment, although the neural network is tailored to suit theparticular purpose of the diagnostic support. However, since the neuralnetwork is composed of software in the embodiment, so in actuality thecoefficients for all diagnostic supports are held, and the diagnosticsupport coefficients for a diagnostic support to be performed are set.

In the utilization phase, the internal coefficients that have beenlearned are set in the neural network in accordance with the purpose ofthe diagnosis. Thereafter, the determination results specify the threeparameters for unknown shadows and the neural network output isobtained. If the neural network has been programmed to output “1” for apositive result and “0” for a negative result, then an output of “0.5”or more is considered positive and an output of less than “0.5” isconsidered negative, and the appropriate determinations are output. Acircular pattern (shadow) determined to be positive is then used in apositive area determination process of a step 34.

The purpose of the positive area determination process in the step 34 isto merge a plurality of adjacent shadow patterns deemed positive into asingle positive area, and to form such positive area into a normalizedrectangle for the purpose of designating an encoded area. It should benoted that a positive area need not be identified as a normalizedrectangle but may be designated with some other shape, so that, forexample, if using MAXSHIFT with JPEG2000, it is possible to specify thatthe area of interest (the positive area) be circular.

As described above, FIGS. 7A, 7B, 7C and 7D are diagrams illustrating ahypothetical example of the shadow-based diagnostic support process.FIG. 7A shows a frontal chest image, which is input as image data 513.FIG. 7B shows the results of the application of the shadow extractionprocess of step S31 to the image. The pathological parameter extractionprocess of step S32 then extracts pathological parameters for each ofthe (in this case) four shadow candidates shown in FIG. 7B. Next, instep S33, the neural network uses the extracted pathological parametersto determine whether the shadow candidates are positive or negative. Thehypothetical case shown in FIG. 7C. indicates a situation in which theneural network determines that three of the four shadow candidates arepositive while the fourth candidate is deemed negative and dropped fromthe display. Finally, FIG. 7D shows the positive shadow patternsresulting from the foregoing image processing, with the additional stepof amalgamating the two shadows in the left lung (shown as the rightlung in the diagram) into a single positive area.

It should be noted that the prerequisites for such a merge process of aplurality of positive areas into a single positive area as isillustrated here is that any given positive area be within a certainrange of another positive area.

As an example of an area merge algorithm, after a certain number ofmorphological dilations have been carried out the number of individuallabeling areas is counted. If the number of individual areas decreasesthen it is determined that a conjunction has taken place. A region ofinterest (hereinafter sometimes referred to as an ROI) is then set so asto encompass the original positive areas in the conjoined region, whichROI is then output as FIG. 7D.

A description will now be given of the texture-typed disease diagnosticsupport, using the example of detection of interstitial pneumonia in afrontal chest image for purposes of illustration only.

FIG. 8 is a flow chart illustrating steps in a texture diseasediagnostic support process.

FIG. 9 is a block diagram of a texture disease diagnostic support unit.FIG. 10 is a block diagram of a segmentation unit of an image processingapparatus according to the present invention. FIG. 11 is a diagramshowing the functional composition of the segmentation unit during atraining phase. FIG. 12 is a diagram showing the functional compositionof the segmentation unit during a utilization phase. FIGS. 13A, 13B and13C illustrate a process of segmentation. FIGS. 14A, 14B and 14C arespecific examples of hypothetical X-ray images.

As shown in FIG. 8, in a step S81, texture disease extraction processingis performed on image data 13 output from the decoder unit 12 and atexture disease candidate region of interest (ROI) is extracted. Asshown in FIG. 9, the input image data 13 is input to a segmentation unit91 and the lung region is segmented. With interstitial disease, areas ofmediastinal space are not of interest, and unlike shadow patterndetection is determined on the basis of localized areas. Accordingly,the lung image must be extracted in such a way as to show local areas ofthe lung.

A more detailed description will now be given of the segmentation unit91, with reference to FIG. 10, which, as described above, is a blockdiagram of a segmentation unit of an image processing apparatusaccording to the present invention. In FIG. 10, reference numeral 101denotes a segment parameter extraction unit, which calculates theparameters for each pixel of the image data 13. Reference numeral 102denotes a discriminator part, which uses a neural network to identify adisease candidate area.

The composition of the segmentation unit 91 is different for thetraining phase and the utilization phase. The segmentation unit 91according to the present embodiment consists of a neural network, so aphase in which practice data is input and the neural network internalcoefficients are formed is called the training phase and the phase inwhich input image data undergoes segmentation is called the utilizationphase. Although in the phases discussed below segmentation is carriedout in pixel units, those of ordinary skill in the art can appreciatethat the present invention is not limited to such a segmentationtechnique but can be adapted to other segmentation techniques, such as,for example, tracing the contour of the image. A detailed description ofsuch contour tracing is provided in 0. Tsujii, M. T. Freedman and S. K.Mun, “Lung Contour Detection in Chest Radiographs Using I-D ConvolutionNeural Networks,” Electronic Imaging, 8(1) (January 1999), pp. 46–53.

As noted above, FIG. 11 is a diagram showing the functional compositionof the segmentation unit during a training phase. A segment parameterextraction unit 101 calculates parameters for each of the pixels of aninput image. The parameters so calculated may be based on either thepixel value, the texture or the relative address from the anatomicalstructure. A more detailed explanation can be found in O. Tsujii, M. T.Freedman and S. K. Mun, “Automated Segmentation of Anatomic Regions inChest Radiographs Using Adaptive-Sized Hybrid Neural Networks,” Med.Phys., 25(6), pp. 46–53, June 1998.

Those of ordinary skill in the art can appreciate that the parameters tobe used are not limited to those disclosed in the above-describedexamples and articles, and that the present invention can be adapted touse other probabilistic parameters such as those described in, forexample, F. McNitt-Gray, H. K. Huang and J. W. Sayre, “Feature Selectionin the Pattern Classification Problem of Digital Chest RadiographSegmentation,” IEEE Trans. Med. Imag., 14, 537–547 (1995).

Next, a description will be given of the discriminator part 102(hereinafter also referred to as the neural network). As describedabove, a variety of systems have been developed for the discriminatorpart 102 neural network. The present embodiment uses an RBF-NN, adaptingthe 3-layer structure of the Feed Forward type and, as shown in FIG. 6and described above, has an input layer, an intermediate layer and anoutput layer. For convenience, there are three input nodes, fourintermediate neurons and one output node.

The input layer has as many nodes as the number of parameters extractedby the segment parameter extraction unit 101. The RBF neurons of theintermediate layer are equipped with output characteristics so as tohave a Gauss distribution as non-linear-type elements. The number of RBFneurons depends on the number of learning cases and the complexity ofthe problem, but in order to set the calculation time to a reasonableperiod 5,000 is a reasonable number. The number of outputs of the neuralnetwork corresponds to the number of segments for each part. Thus, forexample, a frontal chest image such as that shown in FIG. 13A has twoanatomical segments such as those shown in FIG. 13B, and consequentlytwo output nodes are provided.

In the present embodiment, though the discriminator part 102 of neuralnetwork is arranged for every body part (that is, each object image).However, the neural network is composed of software, so in actuality thecoefficients for all body parts are held and the coefficients necessaryfor each body part are set at activation.

In the training phase, parameters for each pixel as well as sampleanswers (teaching output images) that are segmentation results arepresented to the discriminator part 102 neural network. The teachingoutput image for the input image is provided from a segmentationprocessing unit 111 (see FIG. 11) in which the segmentation is carriedout by human intervention. Thus, the teaching image that corresponds tothe input chest image of FIG. 13A is shown in FIG. 13B. In order toaccommodate the segment classification that corresponds to the frontalchest image, the segmentation processing unit 111 produces a teachingimage having output values “1” and “2”. Those areas that are nottargeted for segmentation are assigned other numbers besides “1” and“2”. Such other number may be zero. The production of the teaching imageby the segmentation processing unit 111 can be carried out by computergraphics software capable of specifying an area within the image. Suchgraphics software may be the commercially available “Photoshop” computergraphics from Adobe Corp. of the United States. Neural network learning,if carried out using the least square method to minimize output error,can be obtained analytically and the internal coefficient calculationcan be carried out quickly.

Next, a description will be given of the utilization phase.

As described above, FIG. 12 is a diagram showing the functionalcomposition of the segmentation unit during a utilization phase.

The extraction of parameters for the pixels of the image by the segmentparameter extraction unit 101 is carried out in the same manner as forthe training phase. The internal coefficient corresponding to thediscriminator part 102 neural network is loaded prior to use of thecoefficient corresponding to the portion to be photographed. Theparameters corresponding to each pixel are presented to the neuralnetwork and the neural network outputs an output value to the networkoutput node. In the case of a frontal chest image, there are two outputnodes, with the input pixels being classified to the segment thatcorresponds to the node that outputs the nearest to YES. It should benoted that the coefficient so loaded is acquired by learning by theneural network for each body part. This coefficient originates from therelevant initial values, is matched with a particular photographicportion, a particular diagnostic support, and converged during thelearning process. This converged coefficient determines the behavior ofthe neural network, and typically consists of a plurality ofcoefficients. It should be noted that the number of coefficients dependson the type and size of the neural network used.

A sample segment image for which classification of all the pixels in animage has been completed in the manner described above is shown in FIG.13C. In general, the segment image that the discriminator part 102neural network outputs includes noise. For example, a small surface areaof the lung region can become separated from the lung region. This sortof noisy small area can be deleted during downstream processing, adetailed discussion of which is included in the previously mentionedreference, 0. Tsujii, M. T. Freedman and S. K. Mun, “Lung ContourDetection in Chest Radiographs Using I-D Convolution Neural Networks,”Electronic Imaging, 8(1) (January 1999), pp. 46–53.

The output of the discriminator part 102 neural network of FIG. 10 is asegmentation output image 103. In the case of a chest frontal image thisis the lung portion, a sample of which is shown in FIG. 13C.

Next, an area division process 92 (see FIG. 9) is applied to theabove-described segmentation output image 103. The area division processis a process that defines a local area for the purpose of calculatingtexture disease parameters, a specific example of which is depicted inFIGS. 14A, 14B and 14C.

FIGS. 14A, 14B and 14C are specific examples of hypothetical X-rayimages. More specifically, FIG. 14A is a chest frontal image, which isinput as image data 13 by the segmentation unit 91. The segmentationunit 91 outputs the input image as the segmentation output image 103shown in FIG. 14B. Applying the area division process 92 to thesegmentation output image 103 produces the area-divided image shown inFIG. 14C. That is, the division process divides the target area of thesegmentation output image (in this case the lungs) into normalizedrectangle sections so that the image acquires a surface area at or belowa certain value. A substantially normalized rectangle region of interestdefined in such a manner image data 13 called a texture diseasecandidate region of interest (ROI) 93.

As described above, the output of the texture disease extractionprocessing of step S81 shown in FIG. 8 is a texture disease candidate(ROI) 93. Pathological parameter extraction is performed in a step S82on each of the plurality of texture disease candidate ROI 93 todetermine whether each of the disease candidate is positive or negative.

The processes downstream of parameter calculation are the same as theshadow diagnostic support process depicted in FIG. 3. That is,normalized rectangle positive regions that include texture-typed diseasecandidate ROI deemed positive as shown by the hatching in FIG. 14C aredefined in steps S83 and S84. The positive regions are then output in astep S85. However, as noted previously, parameter calculation is carriedout in accordance with the purpose of the diagnostic support. Specificparameters used for texture-typed disease diagnostic support aredescribed for example in Shigehiko Katsurgawa et al., “The Possibilitiesfor Computer Diagnostic Support for Interstitial Disease”, JapanRadiological Society Journal 50: 753–766 (1990), and Yasuo Sasaki etal., “Quantitative Evaluation by Texture Analysis of PneumoconiosisReference Photographs”, Japan Radiological Society Journal 52: 1385–1393(1992).

When, as described above, the diagnostic support unit 16 outputspositive areas 17, such output is input to the input control unit 14 andused to control the image input fro the image input unit 11. The purposeof the input control unit 14 is to read the data for the positive areas17 (hereinafter also sometimes referred to as regions of interest orROI) in order to improve the detail of the image of the ROI and thusimprove the accuracy of the diagnostic support, or, put another way, tocontribute to the display of a more detailed image of the ROI. Whenimproving the accuracy of the diagnostic support, the diagnostic supportunit 16 is again applied to the accuracy-improved input image. Adescription of the input control unit 14 is given below.

The input control unit 14 designates a tile region or a code blockregion for an ROI and instructs an image server to expedite transfer ofthe corresponding data. The selective data request to the image serveris performed via the image input unit 11, and depending on theconfiguration of the image server, it is possible to store all codeslossless and to form a layered stream dynamically in response to arequest from the image input unit 11.

FIG. 15 is a diagram illustrating a process of dynamically generating acode sequence between an image input unit and an image server.

As shown in FIG. 15, the image input unit 11 transmits a request totransmit a position and relevant portion of a ROI within an image to animage server 151. The image server 151 then transmits to the image inputunit 11 the necessary portions from a pre-encoded sequence stored in anexternal storage device 152. A description is given below of outputs ofcode sequences for ROI produced by the forgoing description.

FIG. 16 is a diagram illustrating a process of code sequencetransmission where an image is divided into a tile and encoded.

As shown in FIG. 16, the image is divided into a tile of normalizedrectangle blocks, with each normalized rectangle tile block beingindependently encoded and stored in the external storage device 152. Thetransmission request from the image input unit 11 is input to a parser161 inside the image server 151. Using the input ROI locationinformation, the parser 161 reads and outputs the code sequence for thetile block that corresponds to the ROI from the external storage device152.

It should be noted that, as described above, the tile block codesequences consist of a plurality of layers. In the illustrative exampledepicted in FIG. 16, there are ten layers, number 0 through 9.Accordingly, the output tile block code sequences may consist of encodeddata solely of the required layer or of all the layers. The output codesequences are then collected into a single code sequence, into whichmarker codes are inserted at the head of the tile block encoded data toindicate the beginning of a tile or a tile number.

If no encoded sequences at all have been transmitted prior to the timeof the state shown in FIG. 16, then the input control unit 14 instructsthe image input unit 11 to input a data stream in accordance with theinformation volume set by the initial stream setting unit 18. By sodoing, the image input unit 11 issues a transmission request to theparser 161 to input layers 9, 8 of all tile blocks. The parser 161composes layers 9, 8 of the encoded data of all blocks from block 1though block 16 into a code sequence and outputs that code sequence.Depending on the results of the analysis performed by the diagnosticsupport unit 16, if, for example, the requested region (region ofinterest) is block 5, then the input control unit 14 instructs the imageinput unit 11 to input block 5 first (priority input), causing the imageinput unit 11 to issue a further transmission request for block 5. Oncesuch a request is received, the parser 161 then assemblies all theremaining layers (in this case layers 7 through 0) of block 5 into acode sequence that it then outputs. As a result, all the layers of block5, the block designated as the requested region of interest, aretransmitted while only the necessary minimum number of layers of otherblocks is transmitted. Moreover, if the coding is performed reversibly,then the decoded image is of the same picture quality as the originalfor the ROI block 5 while images of a picture quality adequate forviewing are reproduced for the other blocks.

Those of ordinary skill in the art can appreciate that other, differentmethods of forming the code sequence can be used, as is described belowwith reference to FIG. 17.

FIG. 17 is a diagram illustrating a process of code sequencetransmission where an image has been coded using DWT.

In JPEG2000, the coefficient or quantization index of the sub-bandproduced by DWT is bit-plane encoded in code block units. In the presentillustrative example, the encoded data for the code blocks is stored inthe external storage device 152 in a form not composed of encodedsequences.

For the encoded data of the code blocks stored in the external storagedevice 152 shown in FIG. 17, transmission of the encoded data portionset by the initial stream setting unit 18 has already taken place inaccordance with the purpose of the diagnosis. For example, datacorresponding to a predetermined bit plane or layer from the top hasalready been transmitted by the initial stream setting unit 18, and thediagnostic support unit 16 is using the image decoded from thetransmitted encoded data to begin diagnostic support processing.

FIG. 18 is a diagram illustrating a hypothetical layering of a JPEG2000code sequence. FIG. 19 is a diagram illustrating encoded data arrangedin layers. FIG. 20 is a block diagram of an image processing apparatusaccording to an embodiment of the present invention. FIGS. 21A and 21Bare diagrams illustrating the relation between code block and region ofinterest.

Next, where the ROI is output by the diagnostic support unit 16 as shownin FIG. 21A, the input control unit 14 designates code blocks thatincludes regions of interest as shown in FIG. 21B. In accordance withthe designated code blocks, the image input unit 11 outputs atransmission request to a parser 171 to transmit additional encoded datafor these designated ROI blocks.

In response to the transmission request from the image input unit 11,the parser 171 composes the encoded data of the code blocks of theregions of interest into encoded sequences for output. For selected codeblocks such as those shown in FIG. 21B, all encoded data is transmittedand decoded together with the data previously transmitted so that animage having high picture quality for the ROI can be composed.

As an illustration of the operations described above, in the case oflayer 9 of the encoded sequence shown in FIG. 18, only the encoded datafor the lower bit planes not previously transmitted would be formed intoa new encoded sequence and transmitted. Or, alternatively, by formingand simultaneously transmitting an added layer composed from code blocksadjacent the ROI, it is possible to obtain improved picture quality forimages of the ROI and adjacent areas.

Thus, as described above, the parser 171 generates and outputs anencoded sequence, in which, as shown in FIG. 18, the encoded data of thecode block for the ROI are positioned in the top layer in response to atransmission request from the image input unit 11. By thus holdingencoded data of an intermediate state prior to formation of the encodedsequence in the external storage device 152, the present invention candynamically generate the appropriate encoded sequences in response to anexternal request. Accordingly, even where the tile blocks are not formedas described above, it is possible to generate and output encodedsequences of high picture quality for the ROI in response to a requestfrom the image display side.

A supplementary explanation of the layers in JPEG2000 will now be addedby way of a coding example.

In JPEG2000, by virtue of its hierarchical encoded data structure whichis called layer structure region of interest described above can beencoded first, on a priority basis. FIG. 18, described previously, showsthe state of layering at this time. As shown in FIG. 18, code blocks 3and 4 include the ROI. Based on the input ROI information, an encodedsequence composer 5 then composes an encoded sequence including layer 9which includes only encoding of the code blocks included in the ROI.

By setting the progressive form of the Layer-Resolutionlevel-Component-Position of JPEG2000, the positions of the layers forall the encoded sequences assumes the form shown in FIG. 19, which, asdescribed above, is a diagram illustrating encoded data arranged inlayers, in which only the encoded data of the region of interest isincluded in the layer 9 positioned at the head of the sequence.

Additionally, by not quantizing during the quantization phase of theencoding process and by forming a predetermined upper layer so as tobecome an encoded data amount that corresponds to a compression ratiodetermined in advance, as shown in FIG. 19 the compression ratio can bematched to the layers so as to enable reversible encoding overall.However, for the uppermost layer (that is, for layer 9), in order toinclude only the ROI the encoded data length differs as the size of theregion differs.

Medical images are large, so every detail of the image cannot bedisplayed for observation o a screen with a limited number of pixels. Itis of course possible to determine an area of interest and to enlargefor display just that area, but it is difficult to determine such anarea of interest from a small, not very detailed image and so a meansfor determining such areas of interest automatically instead of byhumans is required. The embodiments described above automaticallyprovide high-resolution, detailed display using areas or regions ofinterest as determined by diagnostic support techniques.

That is, according to the above-described embodiments, diagnosticsupport for medical images can be carried out using the minimum requiredamount of data, without reading all the images, and moreover, detailedimages of portions deemed positive by the diagnostic support means canbe selectively acquired. As a result, efficient data transmission can becarried out and detailed images of important portions can be promptlydisplayed.

Additionally, according to the above-described embodiments, an initialimage of a size and level of detail that is suited to the diagnostic useof the image can be acquired. Thus, for example, in the case of adiagnosis requiring only a rough image the present invention assuresthat no unneeded data transmission occurs, thereby reducing networktraffic and ultimately saving both time and expense.

It should be noted that, where the diagnostic support means shows noareas of concern, an area of interest can be formed by deleting allareas of an irradiated field in which the radiation has passed throughthe field unchecked, as described previously with respect to anapplication already filed by the inventors. Additionally, bydeliberately failing to set (display) an area of interest, it ispossible to provide the attending physician with a clean, that is, anunmarked, image. The advantage of which is that the physician is neitherprejudiced by nor (perhaps misleadingly) sensitized to any given area orareas of the image so supplied.

Those of ordinary skill in the art can appreciate that the objects ofthe present invention are also achieved by supplying a recording mediumon which is recorded a software program code for realizing the functionsof the above-described embodiments to a system or apparatus, and causinga computer, which may be a CPU or MPU, of such system or apparatus toread and execute the program code stored in the recording medium.

In such a case, in which the program code, as read from the recordingmedium, achieves the effects of the above-described embodiments, therecording medium itself constitutes the present invention.

Similarly, in such a case, a program code as described above, so as toachieve the effects of the above-described embodiments, itselfconstitutes the present invention.

Similarly, in such a case, a general-purpose computer, programmed by theabove-described program code to become a special-purpose computerachieving the capabilities of the above-described embodiments, itselfconstitutes the present invention.

A variety of media may be used as the recording media for supplying theprogram code. These recording media may for example be a floppy disk, ahard disk, an optical disk, a magnetic optical disk, a CD-ROM, a CD-R,magnetic tape, a non-volatile memory card or a ROM.

Those of ordinary skill in the art can appreciate that the effects ofthe above-described embodiments can be achieved not only by causing acomputer to execute the read program code but also includes a case inwhich an operating system (OS) loaded into the computer carries out someor all of the actual processing based on instructions of the programcode, such that the effects of the above-described embodiments areachieved by such processing.

Further, those of ordinary skill in the art can appreciate that theeffects of the present invention can also be achieved by the programcode read from the recording medium, after being written to a memory ina function expansion board inserted in the computer or a functionexpansion unit connected to the computer, causing the CPU or the like ofsuch expansion board or expansion unit to carry out some or all of theactual processing based on the program code instructions, such that theeffects of the above-described embodiments are achieved by suchprocessing.

Thus, as described above, the present invention makes it possible topriority display an area useful to a diagnosis from encoded image data.

Additionally, present invention makes it possible to priority read anarea of interest (AOI) important to the diagnosis or observation asdetermined by a diagnostic support means and to improve the quality ofthat image of that area, so as to effectively improve the accuracy ofdiagnostic support.

Additionally, the present invention makes it possible to acquire imagedata of an image quality suited to the objective of the diagnosticsupport and to achieve efficient image transmission.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificpreferred embodiments described above thereof except as defined in theclaims.

1. An apparatus having an image processing function, comprising: firstacquisition means for acquiring a first portion of data as a data streamobtained from image data that has been sequentially converted andencoded; decoding means for decoding the data acquired by said firstacquisition means and obtaining a two-dimensional image; analysis meansfor analyzing the two-dimensional image obtained by said decoding meansand determining a region of interest within the two-dimensional image;and second acquisition means for acquiring a second portion of data as adata stream obtained from the image data based on the region of interestdetermined by said analysis means, wherein the apparatus causes saiddecoding means and said analysis means to operate on the basis of thedata stream acquired by said second acquisition means.
 2. A method ofimage processing, comprising: a first acquisition step of acquiring afirst portion of data as a data stream obtained from image data that hasbeen sequentially converted and encoded; a decoding step of decoding thedata acquired in the first acquisition step and obtaining atwo-dimensional image; an analysis step of analyzing the two-dimensionalimage obtained in the decoding step and determining a region of interestwithin the two-dimensional image; and a second acquisition step ofacquiring a second portion of data as a data stream obtained from theimage data based on the region of interest determined in the analysisstep, wherein the decoding step and the analysis step operate on thebasis of the data stream acquired in the second acquisition step.